How do I grab all the links within an element in HTML using python? -

-

first, please check image below can better explain question:

enter image description here

i trying take user input select 1 of links below "course search term".... (ie. winter 2015).

the html opened shows part of code webpage. grab href links in element , consists of 5 term links want. following instructions website (www.gregreda.com/2013/03/03/web-scraping-101-with-python/), doesn't explain part. here code have been trying.

from bs4 import beautifulsoup urllib2 import urlopen  base_url = "http://classes.uoregon.edu/"  def get_category_links(section_url):      html = urlopen(section_url).read()     soup = beautifulsoup(html, "lxml")     pldefault = soup.find("td", "pldefault")     ul_links = pldefault.find("ul")     category_links = [base_url + ul.a["href"] in ul_links.findall("ul")]      return category_links

any appreciated! thanks. or if see website, classes.uoregon.edu/

i keep simple , locate links containing 2015 in text , term in href:

for link in soup.find_all("a",                           href=lambda href: href , "term" in href,                           text=lambda text: text , "2015" in text):     print link["href"]

prints:

/pls/prod/hwskdhnt.p_search?term=201402 /pls/prod/hwskdhnt.p_search?term=201403 /pls/prod/hwskdhnt.p_search?term=201404 /pls/prod/hwskdhnt.p_search?term=201406 /pls/prod/hwskdhnt.p_search?term=201407

if want full urls, use urlparse.urljoin() join links base url:

from urlparse import urljoin  ... link in soup.find_all("a",                           href=lambda href: href , "term" in href,                           text=lambda text: text , "2015" in text):     print urljoin(url, link["href"])

this print:

http://classes.uoregon.edu/pls/prod/hwskdhnt.p_search?term=201402 http://classes.uoregon.edu/pls/prod/hwskdhnt.p_search?term=201403 http://classes.uoregon.edu/pls/prod/hwskdhnt.p_search?term=201404 http://classes.uoregon.edu/pls/prod/hwskdhnt.p_search?term=201406 http://classes.uoregon.edu/pls/prod/hwskdhnt.p_search?term=201407

Comments

Post a Comment

Popular posts from this blog

node.js - Mongoose: Cast to ObjectId failed for value on newly created object after setting the value -

-
this question has answer here: what's mongoose error cast objectid failed value xxx @ path “_id”? 7 answers even though has been asked many times because entity has no valid objectid field. i came across situation objectid set value can casted objectid still error. the schema: var serviceschema = parammongoose.schema({ servicename:{ type: parammongoose.schema.types.objectid , ref: 'servicenames',required:true } }); pseudo code: servicefromdb=new service({servicename:'some name'}); servicefromdb.servicename='000000000000000000000001'; servicefromdb.save(function(paramerror,paramdata){ if(paramerror){ console.log('but but...',servicefromdb,paramerror) } }); the output of code is: but but... { servicename: 000000000000000000000001, _id: 55079a90286f49280364f78b } { [casterror: cast objectid faile...
Read more

gradle error "Cannot convert the provided notation to a File or URI" -

-
i have defined 2 system properties in gradle.properties: systemprop.builddir=build systemprop.cfgdir=build\\cfg and have following task defined in build.gradle: task clean(group:'clean',description:'clean project') << { ant.sequential { delete(dir: system.properties.builddir) mkdir(dir: system.properties.builddir) delete(dir: system.properties.cfgdir) mkdir(dir: system.properties.cfgdir) } } this generates following error: execution failed task ':clean'. > cannot convert provided notation file or uri: {dir=build}. following types/formats supported: - string or charsequence path, e.g 'src/main/java' or '/usr/include' - string or charsequence uri, e.g 'file:/usr/include' - file instance. - uri or url instance. but equivalent block of code not generate error , works expected: task clean(group:'clean',description:'clean project') << { ...
Read more

python - NameError: name 'subprocess' is not defined -

-
i'm trying make rng website: http://jamesdotcom.com/?p=417 . when run following code: captureimage = subprocess.popen( [ "fswebcam", "-r", "356x292", "-d", "/dev/video0", "static.jpg", "--skip", "10" ], stdout=devnull, stderr=devnull) captureimage.communicate() i following error message: nameerror: name 'subprocess' not defined what should do? thanks have imported first? import subprocess captureimage = subprocess.popen(["fswebcam", ...) subprocess included in python standard library since python 2.4. should not necessary install it.
Read more