import requests
from bs4 import BeautifulSoup

res = requests.get("https://mctopherganesh.com")
print(res)

<Response [200]>

soup = BeautifulSoup(res.text)

soup.title.text

'Welcome Muggles'

a_tags = soup.find_all('a')

for tag in a_tags:
    print(tag.get('href'))

index.html
index.html
blog.html
about.html
https://github.com/Midnight-Lizard/Midnight-Lizard
#collapseExample
#collapseExample

def gather_homepage_links():
    
    import requests
    from bs4 import BeautifulSoup
    
    link_names = []
    
    response = requests.get("https://mctopherganesh.com")
    soup = BeautifulSoup(response.text)
    for tag in soup.find_all('a'):
        link_names.append(tag.get('href'))
        
    return link_names

print(gather_homepage_links())

['index.html', 'index.html', 'blog.html', 'about.html', 'https://github.com/Midnight-Lizard/Midnight-Lizard', '#collapseExample', '#collapseExample']

homepage_links = gather_homepage_links() 
homepage_links[2]

'blog.html'

blog_suffix = homepage_links[2]
blog_page = requests.get("https://mctopherganesh.com/" + blog_suffix)
print(blog_page)

<Response [200]>

blog_html = BeautifulSoup(blog_page.text)
blog_links = []
for tag in blog_html.find_all('a'):
    blog_links.append(tag.get('href'))

blog_links

['index.html',
 'index.html',
 'blog.html',
 'about.html',
 '/blog/day19.html',
 '/blog/day18.html',
 '/blog/day17.html',
 '/blog/day16.html',
 '/blog/day013_014_and_015.html',
 '/blog/day012_day013_maybe.html',
 '/blog/day011.html',
 '/blog/day010.html',
 '/blog/why_does_posting_take_so_long.html',
 '/blog/day08.html',
 '/blog/day07.html',
 '/blog/day06.html',
 '/blog/day05.html',
 '/blog/day04.html',
 '/blog/day03ish.html',
 '/blog/day02.html',
 '/blog/day1again.html',
 '/blog/day15.html',
 '/blog/day13_and_14.html',
 '/blog/day12.html',
 '/blog/day11.html',
 '/blog/day10.html',
 '/blog/day7_and_8_and_9.html',
 '/blog/day6.html',
 '/blog/day5',
 '/blog/day4.html',
 '/blog/day3.html',
 '/blog/day2.html',
 '/blog/day1.html']

welcome to day 20 and 21 stalkers.

what's all of that up there?

believe or not, I wrote all of that code because I was frustrated. I was frustrated that I can't just decide how to build a fancy shmancy website and build it as I go.

Websites must be planned out. This probably seems obvious to you but to me, coding has always been this thing I can just do and when I run into obstacles I kind of figure out my way over, under, or around them. Maybe through them, it's all about perspective you see.

However with websites, you have to know what your website is going to look like BEFORE you actually code it up.

That's craziness. I coded all of the above because I felt like coding without direction and my favorite kind of coding is scraping.

What's the real problem?

The real problem is I have spent years on learning how to code in Python, especially web scraping (it's my pb and jam) but I have NOT spent years developing websites which means I have done everything I can to avoid planning when it comes to programming.

I like the doing. Not the planning and doing.

Not a huge problem though. Pretty funny realization to have. It's not truly something I didn't know to be honest, but it was one of those things I knew and didn't want fully manifest what it meant, you know?

Oh, you know what I'm talking about.

Yesterday I spent a good amount of time working on Dataquest. Learning some pretty neat things on there. Messing with the pandas library. Good stuff.

Today I sat down to mess with my website and had the above realization that I'm not at a place where I can simply hack together a website. The code above? It's not a perfect webscrape by any means but I have enough experience writing python scripts and building them out into fully fledged...scripts/programs/code..stuff.

It is a little inspiring to know that I have a preference coding with Python to making websites because I can slap together a python program decently enough and I cannot slap together a website. I do remember when I couldn't do any slapping of any kind. Solely knowing that is kind of cool right?