Website photo with search box visible.
So, this is the website
https://sa.ucla.edu/ro/public/soc
There is a dropdown menu for selecting subject area where I need to write subject and i will receive results that I need.
I want to automate this using selenium.
Turns out, that particular part of the code in bs4 (before js loads) is a script. This is the container element I printed using bs4
<div id="partial_content">
<div class="positionRelative"><div class="hide" id="div_infomsg_searchPanel"><img alt="info icon" height="17" src="/ro/iwe/img/info_icon.gif" width="17"><span id="searchPanelInfoMsg"></span></img></div></div>
<div id="div_subjectArea">
<div class="ui-widget hide" id="div_subject">
<iwe-autocomplete class="search_input" clear_on_blur="" clear_on_focus="" id="select_filter_subject" input_class="parentSpanning" label_class="fakeHidden" name="SubjectAreaName" show_all_options="" visible_label_id="select_filter_subject_label"></iwe-autocomplete>
<input id="subject_area" name="subj" type="hidden">
</input></div>
<div class="ui-widget hide" id="div_catalog">
<iwe-autocomplete class="search_input" clear_on_blur="" clear_on_focus="" id="select_filter_catalog" input_class="parentSpanning" label_class="fakeHidden" name="CrsCatlgName" show_all_options="" visible_label_id="select_filter_catalog_label"></iwe-autocomplete>
<input id="catalog" name="catlg" type="hidden"/>
</div>
<input id="subjectArea_classNo" name="cls_no" type="hidden"/>
</div>
<div class="control" id="div_btn_go"><input class="button hide" id="btn_go" type="submit" value="Go"/></div>
<script type="text/javascript">
$(document).ready(function () {
SearchPanelSetup('[{"label":"Aerospace Studies (AERO ST)","value":"AERO ST"},{"label":"African American Studies (AF AMER)","value":"AF AMER"},{"label":"American Indian Studies (AM IND)","value":"AM IND "},{"label":"American Sign Language (ASL)","value":"ASL "},{"label":"Ancient Near East (AN N EA)","value":"AN N EA"},{"label":"Anthropology (ANTHRO)","value":"ANTHRO "},{"label":"Applied Chemical Sciences (APP CHM)","value":"APP CHM"},{"label":"Arabic","value":"ARABIC "},{"label":"Archaeology (ARCHEOL)","value":"ARCHEOL"},{"label":"Architecture and Urban Design (ARCH\u0026UD)","value":"ARCH\u0026UD"},{"label":"Armenian (ARMENIA)","value":"ARMENIA"},{"label":"Art","value":"ART "},{"label":"Art History (ART HIS)","value":"ART HIS"},{"label":"Arts Education (ARTS ED)","value":"ARTS ED"},{"label":"Asian","value":"ASIAN "},{"label":"Asian American Studies (ASIA AM)","value":"ASIA AM"},{"label":"Astronomy (ASTR)","value":"ASTR "},{"label":"Atmospheric and Oceanic Sciences (A\u0026O SCI)","value":"A\u0026O SCI"},{"label":"Bioengineering (BIOENGR)","value":"BIOENGR"},{"label":"Bioinformatics (Graduate) (BIOINFO)","value":"BIOINFO"},{"label":"Biological Chemistry (BIOL CH)","value":"BIOL CH"},{"label":"Biomathematics (BIOMATH)","value":"BIOMATH"},{"label":"Biomedical Research (BMD RES)","value":"BMD RES"},{"label":"Biostatistics (BIOSTAT)","value":"BIOSTAT"},{"label":"Central and East European Studies (C\u0026EE ST)","value":"C\u0026EE ST"},{"label":"Chemical Engineering (CH ENGR)","value":"CH ENGR"},{"label":"Chemistry and Biochemistry (CHEM)","value":"CHEM "},{"label":"Chicana/o and Central American Studies (CCAS)","value":"CCAS "},{"label":"Chinese (CHIN)","value":"CHIN "},{"label":"Civil and Environmental Engineering (C\u0026EE)","value":"C\u0026EE "},{"label":"Classics (CLASSIC)","value":"CLASSIC"},{"label":"Clusters (CLUSTER)","value":"CLUSTER"},{"label":"Communication (COMM)","value":"COMM "},{"label":"Community Engagement and Social Change (CESC)","value":"CESC "},{"label":"Community Health Sciences (COM HLT)","value":"COM HLT"},{"label":"Comparative Literature (COM LIT)","value":"COM LIT"},{"label":"Computational and Systems Biology (C\u0026S BIO)","value":"C\u0026S BIO"},{"label":"Computer Science (COM SCI)","value":"COM SCI"},{"label":"Conservation of Cultural Heritage (CLT HTG)","value":"CLT HTG"},{"label":"Dance","value":"DANCE "},{"label":"Data Science in Biomedicine (DS BMED)","value":"DS BMED"},{"label":"Design / Media Arts (DESMA)","value":"DESMA "},{"label":"Digital Humanities (DGT HUM)","value":"DGT HUM"},{"label":"Disability Studies (DIS STD)","value":"DIS STD"},{"label":"Dutch","value":"DUTCH "},{"label":"Earth, Planetary, and Space Sciences (EPS SCI)","value":"EPS SCI"},{"label":"Ecology and Evolutionary Biology (EE BIOL)","value":"EE BIOL"},{"label":"Economics (ECON)","value":"ECON "},{"label":"Education (EDUC)","value":"EDUC "},{"label":"Electrical and Computer Engineering (EC ENGR)","value":"EC ENGR"},{"label":"Engineering (ENGR)","value":"ENGR "},{"label":"English (ENGL)","value":"ENGL "},{"label":"English as A Second Language (ESL)","value":"ESL "},{"label":"English Composition (ENGCOMP)","value":"ENGCOMP"},{"label":"Environment (ENVIRON)","value":"ENVIRON"},{"label":"Environmental Health Sciences (ENV HLT)","value":"ENV HLT"},{"label":"Epidemiology (EPIDEM)","value":"EPIDEM "},{"label":"Ethnomusicology (ETHNMUS)","value":"ETHNMUS"},{"label":"European Languages and Transcultural Studies (ELTS)","value":"ELTS "},{"label":"Filipino (FILIPNO)","value":"FILIPNO"},{"label":"Film and Television (FILM TV)","value":"FILM TV"},{"label":"Food Studies (FOOD ST)","value":"FOOD ST"},{"label":"French (FRNCH)","value":"FRNCH "},{"label":"Gender Studies (GENDER)","value":"GENDER "},{"label":"Geography (GEOG)","value":"GEOG "},{"label":"German","value":"GERMAN "},{"label":"Gerontology (GRNTLGY)","value":"GRNTLGY"},{"label":"Global Health (GLB HLT)","value":"GLB HLT"},{"label":"Global Jazz Studies (GJ STDS)","value":"GJ STDS"},{"label":"Global Studies (GLBL ST)","value":"GLBL ST"},{"label":"Graduate Student Professional Development (GRAD PD)","value":"GRAD PD"},{"label":"Greek","value":"GREEK "},{"label":"Health Policy and Management (HLT POL)","value":"HLT POL"},{"label":"Healthcare Administration (HLT ADM)","value":"HLT ADM"},{"label":"Hebrew","value":"HEBREW "},{"label":"Hindi-Urdu (HIN-URD)","value":"HIN-URD"},{"label":"History (HIST)","value":"HIST "},{"label":"Honors Collegium (HNRS)","value":"HNRS "},{"label":"Human Genetics (HUM GEN)","value":"HUM GEN"},{"label":"Hungarian (HNGAR)","value":"HNGAR "},{"label":"Indigenous Languages of the Americas (IL AMER)","value":"IL AMER"},{"label":"Indo-European Studies (I E STD)","value":"I E STD"},{"label":"Indonesian (INDO)","value":"INDO "},{"label":"Information Studies (INF STD)","value":"INF STD"},{"label":"International and Area Studies (I A STD)","value":"I A STD"},{"label":"International Development Studies (INTL DV)","value":"INTL DV"},{"label":"International Migration Studies (I M STD)","value":"I M STD"},{"label":"Iranian","value":"IRANIAN"},{"label":"Islamic Studies (ISLM ST)","value":"ISLM ST"},{"label":"Italian","value":"ITALIAN"},{"label":"Japanese (JAPAN)","value":"JAPAN "},{"label":"Korean (KOREA)","value":"KOREA "},{"label":"Labor Studies (LBR STD)","value":"LBR STD"},{"label":"Latin","value":"LATIN "},{"label":"Latin American Studies (LATN AM)","value":"LATN AM"},{"label":"Law","value":"LAW "},{"label":"Lesbian, Gay, Bisexual, Transgender, and Queer Studies (LGBTQS)","value":"LGBTQS "},{"label":"Life Sciences (LIFESCI)","value":"LIFESCI"},{"label":"Linguistics (LING)","value":"LING "},{"label":"Management (MGMT)","value":"MGMT "},{"label":"Management-Executive MBA (MGMTEX)","value":"MGMTEX "},{"label":"Management-Full-Time MBA (MGMTFT)","value":"MGMTFT "},{"label":"Management-Fully Employed MBA (MGMTFE)","value":"MGMTFE "},{"label":"Management-Global Executive MBA Asia Pacific (MGMTGEX)","value":"MGMTGEX"},{"label":"Management-Master of Financial Engineering (MGMTMFE)","value":"MGMTMFE"},{"label":"Management-Master of Science in Business Analytics (MGMTMSA)","value":"MGMTMSA"},{"label":"Management-PhD (MGMTPHD)","value":"MGMTPHD"},{"label":"Materials Science and Engineering (MAT SCI)","value":"MAT SCI"},{"label":"Mathematics (MATH)","value":"MATH "},{"label":"Mechanical and Aerospace Engineering (MECH\u0026AE)","value":"MECH\u0026AE"},{"label":"Microbiology, Immunology, and Molecular Genetics (MIMG)","value":"MIMG "},{"label":"Middle Eastern Studies (M E STD)","value":"M E STD"},{"label":"Military Science (MIL SCI)","value":"MIL SCI"},{"label":"Molecular and Medical Pharmacology (M PHARM)","value":"M PHARM"},{"label":"Molecular Biology (MOL BIO)","value":"MOL BIO"},{"label":"Molecular Toxicology (MOL TOX)","value":"MOL TOX"},{"label":"Molecular, Cell, and Developmental Biology (MCD BIO)","value":"MCD BIO"},{"label":"Molecular, Cellular, and Integrative Physiology (MC\u0026IP)","value":"MC\u0026IP "},{"label":"Music (MUSC)","value":"MUSC "},{"label":"Music Industry (MSC IND)","value":"MSC IND"},{"label":"Musicology (MUSCLG)","value":"MUSCLG "},{"label":"Naval Science (NAV SCI)","value":"NAV SCI"},{"label":"Near Eastern Languages (NR EAST)","value":"NR EAST"},{"label":"Neurobiology (NEURBIO)","value":"NEURBIO"},{"label":"Neuroscience (Graduate) (NEURO)","value":"NEURO "},{"label":"Neuroscience (NEUROSC)","value":"NEUROSC"},{"label":"Nursing","value":"NURSING"},{"label":"Oral Biology (ORL BIO)","value":"ORL BIO"},{"label":"Pathology and Laboratory Medicine (PATH)","value":"PATH "},{"label":"Philosophy (PHILOS)","value":"PHILOS "},{"label":"Physics","value":"PHYSICS"},{"label":"Physics and Biology in Medicine (PBMED)","value":"PBMED "},{"label":"Physiological Science (PHYSCI)","value":"PHYSCI "},{"label":"Polish (POLSH)","value":"POLSH "},{"label":"Political Science (POL SCI)","value":"POL SCI"},{"label":"Portuguese (PORTGSE)","value":"PORTGSE"},{"label":"Program in Computing (COMPTNG)","value":"COMPTNG"},{"label":"Psychiatry and Biobehavioral Sciences (PSYCTRY)","value":"PSYCTRY"},{"label":"Psychology (PSYCH)","value":"PSYCH "},{"label":"Public Affairs (PUB AFF)","value":"PUB AFF"},{"label":"Public Health (PUB HLT)","value":"PUB HLT"},{"label":"Public Policy (PUB PLC)","value":"PUB PLC"},{"label":"Quantum Science and Technology (QNT SCI)","value":"QNT SCI"},{"label":"Religion, Study of (RELIGN)","value":"RELIGN "},{"label":"Research Practice (RES PRC)","value":"RES PRC"},{"label":"Romanian (ROMANIA)","value":"ROMANIA"},{"label":"Russian (RUSSN)","value":"RUSSN "},{"label":"Scandinavian (SCAND)","value":"SCAND "},{"label":"Science Education (SCI EDU)","value":"SCI EDU"},{"label":"Semitic","value":"SEMITIC"},{"label":"Serbian/Croatian (SRB CRO)","value":"SRB CRO"},{"label":"Slavic (SLAVC)","value":"SLAVC "},{"label":"Social Science (SOC SC)","value":"SOC SC "},{"label":"Social Welfare (SOC WLF)","value":"SOC WLF"},{"label":"Society and Genetics (SOC GEN)","value":"SOC GEN"},{"label":"Sociology (SOCIOL)","value":"SOCIOL "},{"label":"South Asian (S ASIAN)","value":"S ASIAN"},{"label":"Southeast Asian (SEASIAN)","value":"SEASIAN"},{"label":"Spanish (SPAN)","value":"SPAN "},{"label":"Statistics (STATS)","value":"STATS "},{"label":"Swahili","value":"SWAHILI"},{"label":"Thai","value":"THAI "},{"label":"Theater","value":"THEATER"},{"label":"Turkic Languages (TURKIC)","value":"TURKIC "},{"label":"Ukrainian (UKRN)","value":"UKRN "},{"label":"University Studies (UNIV ST)","value":"UNIV ST"},{"label":"Urban Planning (URBN PL)","value":"URBN PL"},{"label":"Vietnamese (VIETMSE)","value":"VIETMSE"},{"label":"World Arts and Cultures (WL ARTS)","value":"WL ARTS"},{"label":"Yiddish (YIDDSH)","value":"YIDDSH "}]', 'select_filter_subject', 'Enter Subject Area (Required)', '_Select_Subject');
//originally Subject partial didn't have/need the hide class, but to standardize everything it was added
//so to remove it, use the following js
globalFunction.showElements(["div_subject"]);
});
</script>
</div>
I want to access that search box using selenium.
I used bs4 first but got this output.
Then used selenium, my code looks like this
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# Initialize the WebDriver (e.g., Chrome)
driver = webdriver.Chrome()
driver.get("https://sa.ucla.edu/ro/public/soc")
try:
# Execute JavaScript to get the element from the Shadow DOM
js_executor = driver
shadow_host = driver.find_element(
By.CSS_SELECTOR, "iwe-autocomplete#select_filter_subject"
)
element_in_shadow_dom = js_executor.execute_script(
'return arguments[0].shadowRoot.querySelector("input[type=text]")', shadow_host
)
# Perform action
element_in_shadow_dom.send_keys("Test input")
input("press enter")
print("Successfully interacted with the iwe-autocomplete element.")
finally:
driver.quit()
I used shadow_host because one website recommended me to. I dont really know why. It did not work with normal find element too. Same error: Element not found Exception and timeout exception.
I also tried waiting (not in this one) I waited 30 seconds in one script but still the issue was the same.
try/exceptto get FULL error message. And show this error in question (not in comments) as text (not image)