Saturday, July 13, 2019

Capstone Adventures: Part 2

It took a few tweaks to get the image search and sort scripts working. The process is finally streamlined and I was able to get through 3 of the 14 batches until I stopped to attempt to send some of those files to Azure.

Of course this didn't work, otherwise I wouldn't have had to add an extra post today. Apparently Azure doesn't like .tif images. After confirming that it prefers some other file format, I installed IrfanView to start batch converting them before upload.

The last hiccough of the day was having to stop that process because my computer was running out of space again. So I'm now uploading the files into Azure so I can make room to convert more files. I think I'll finish uploading the files I processed so far today, then tomorrow I can sort, convert, and upload at the same time (in theory) ... get some sort of assembly line going, lol.

Adventure!
at No comments:

Capstone Adventures: Part 1

This is more like Part 3 but I'll catch you up, lol.

My first choice for capstone project was Recursion Cellular Image Classification to disentangle experimental 'noise' from biological signals. This would help researchers understand how drugs interact with human cells which translates to decreasing the cost of treatments and the time it takes to bring new treatments to market. My computer couldn't handle the dataset. There were over 85GB worth of images and I shouldn't have attempted to download it, lol. It took a day to get the zipped file into my computer and partially unzipped before I ran out of room. Then it took another day to get the files off my computer.

My second choice is currently an image recognition application to identify metastatic tissue in histopathologic scans of lymph node sections. At 6 GB, my computer is currently struggling to do things with the dataset (220,006 image files). Right now it needs to parse a list and separate the 'cancer' images from the 'not cancer' images which were not-so-conveniently itemized in a csv file.

I spent the morning separating those files into 14 different folders in the hopes that my computer finds searching through under 20k files each more reasonable than the full dataset. Of course I'm using automation scripts but my computer is a dinosaur so it's still struggling to get by. But I just need to get the data separated and into Azure. After that I can get these images off my computer and handle everything else from the cloud.

The first file is still running by the time I posted this. Remember I have 14 to get though and once I do all that, I need to upload them into Azure.

I am thankful the cloud exists so I can work with datasets like this! I know I need to update my computer. It's on my list, especially because I REALLY want to attempt the Recursion Cellular Image problem once I have enough time to do it properly.
at No comments:

Monday, July 8, 2019

C951 Introduction to Artificial Intelligence

Both of these tasks ended up being largely an exercise in using buzz words for the paper. It's frustrating because it can easily be turned into something more to include machine learning which is required for capstone. I attempted to do that with Task 1 but it was taking too long to put a proper dataset and base case together. I'll have to attempt it again after graduation.
TASK 1: Start with the chapter in Ucertify called Focus: Chatbots between chapters 11 and 12. Find the exercises and hit the submit button to see examples on how to answer various parts of the rubric. For relevant data, I used O*NET which is what WGU uses for our career assessments. The Bureau of Labor Statistics can help find computer careers if you're not finding 5 distinct ones ... If you follow the rubric, this is a really primitive version of the assessments we get in the career center. My best advice is to play with different chat options. Decide your criteria that can be assessed with questions and answers, then build your conversation paths from there.
TASK 2: Start with the chapter in Ucertify called Focus: Robotics and Feature Engineering between chapters 25 and 26. Again, find the exercises and hit the submit button to see examples on how to answer various parts of the rubric. I started by building the environment to get used to the interface and object manipulation. The first thing I did was create an enclosed space to prevent Rob from falling off the testing area. Then I built a few obstacles and ran the simulation to see what Rob can do and how to improve it. I stuck with adding proximity sensors which involved copying the nose sensor and modifying the sensor name and attributes to suit my needs. Then I modified the main rob script (you'll see the nose sensor stuff in there and should be able to trace the code and use it to add the logic needed to make other sensors work). I ended up with three proximity/collision avoidance sensors that allowed rob to make right and left turns (the default only turns left). I also tweaked the speed and other attributes to force it to cover wider areas instead of getting caught in circular paths or not being able to back out of certain corners. This ended up more interesting than task 1 and I plan to make it more interesting in the future before showcasing it on GitHub.
at No comments:

My Tenure at WGU

I did have my eye on two other degrees at WGU (B.S. Math Ed and M.S. Special Ed) but all things considering, I think this will be my last at WGU. Right now I plan to spend some time taking select advanced math courses at Brandman (through Westcott). From there I will either go for the B.S. Mathematics program at Louisiana State University or the M.S. Computer Science program at Georgia Tech ... I no longer have any interest jumping through WGU hoops to get into the math ed program. Their loss.
at No comments:

Friday, June 28, 2019

C195 Software II

I thought I was going to need the rest of my term to finish this one but here I am! Thank you to all my rubber duckies, testers, and Java gurus who took me in and made sure I didn't chase down too many rabbit holes. 
This class is kind of a mess because it was overhauled a little earlier this year so some students are using one schema and the rest have the updated version. That made it easier to complete but a lot of the old tips and info is not relevant so it's easy to get confused when everyone else is talking about something that no longer applies. Even the intro video goes over things in the old version.
Beginners: start with the webinars! You need to set up MySQL to connect the WGU database and then with your java program. There are no handy text guides though I may work on making one after graduation (I'll update this post once I make it). Setting everything up properly was among the more painful parts of the program for me.
I generally find lectures too slow to watch at normal speed so I recorded them in OBS while catching up on some reading and then took a weekend to go through the webinars on my YouTube account (you will not find them there, they were set to private and I deleted them when I'm done). This allowed me to watch them on my iPad at whatever speed I wanted (often 2x). We're not allowed to share webinars because they are considered school property so if you do put them on YouTube for viewing purposes, set them to private because the school doesn't take kindly to non-WGU students getting a hold of their materials.
Here are a list of webinars. If you are not a WGU student, you will not be able to see them. Sorry!
This was my reading list and reference guide, many from other slack students plus a few by me. Students were just adding their own sources at the end of the list so when I got to it, I cleaned out the dead links and categorized them so I can find them easier. I say this to make sure it stays that way, lol.
Program Design
Data Access
Sample Codes for JDBC
Java Topics
And here are some notes I collected from different parts of chatter plus a few notes from me
  • Run the conversion script first (convertToNewDB_ver2.sql), then run the insertion script (dataForNewDB_ver2.sql).
  • You do not have to make a custom calendar for the ‘view calendar by month and week’ requirement;  you can use a Table View and fill it with appointment the data with the option to sort by month and week
  • Make sure that your project includes the JDBC file in the project folder so when you zip the folder to submit it the file will be in there.
  • When submitting make sure that you build specific exception controls for each control on the form. For example it is not good enough to say "Invalid Customer Information". It needs to say "Invalid Phone Number" or whatever control is wrong on the form.
  • 15-minute alert requirement: have the program check if there is an appointment in within 15 minutes of successful login and show an alert for it/them. Set the 'test' account to see the alerts for all users.
  • Everything time Related goes into the DB as UTC. There's code in the repo to help do this, learning how to grab the local time now and turning it into UTC time should be a static method that you can use at any point.
  • The assignment doesn’t explicitly require a paper but you need one to tell the graders where everything is. Copy the rubric and in each section, identify where in the script the requirement can be found and/or how you fulfilled the requirement. For example, in Section J, identify where the log file will be saved and where this part of the code can be found. For the lambda requirement, list the name of the script and line number.
  • CREATE VIEW is allowed (I used them to add customer names to my appointment tables).
Database notes:
  • Some of the primary keys in the tables aren't set to auto increment. Use ALTER TABLE foo MODIFY COLUMN id INT NOT NULL AUTO_INCREMENT; to change it while you're setting up (one less thing to worry about). There is a script that supposedly does it but it didn't work for me so I did it by hand which took a few minutes.
  • Countries and Cities are static and are given in the rubric so adding those should be part of your database setup
  • I added users and customers to the database while setting up because I wanted to be able to test my functions once they were available. I probably went overboard but I wanted to be sure things worked properly.
Time Zone Applications:
Make friends with Timestamp, DateTimeFormatter, LocalDateTime, ZonedDateTime, ZoneId, and ZonedDateTime. If you want to be fancy, you can check out LocalTime, parseInt, and split to turn the 24H time into 12H; this will require some string splicing which I was fortunate to have played with in DSA2. Don't forget to test both AM and PM if you go this route, those parts can mask certain bugs that the graders will gladly find for you, lol. Take this part step-by-step, utilize println statements (coupled with a //TODO to delete them so they're easy to find and comment out when you're done). Read the time stuff in the book, it is probably the most helpful section in there.
Buchalka's Java Masterclass - https://www.udemy.com/java-the-complete-java-developer-course
I'm glad I took the time to go through this one. There were a lot of programming concepts I was just using without fully understanding how it worked. I still don't 'fully' understand it all but it did clear up a lot of cobwebs. If you're not ready for the project after you watch the webinars, these will be worth your time. You can always skip to the parts you know you need work on if you can actually pinpoint your weaknesses.

My Experience:
I spent a week playing with the examples in the webinars and setting up my program environment and design. I also set up the proper folders and scripts so by the time I started watching the webinars, I would throw in rubric notes or maybe a link to a resource. I set up the database before getting around to the programming since that's the one thing I do have experience with. I wrote up the queries I imagined I would need and saved them for a rainy day. Then I played with GUI components to decide how I wanted everything to look (this changed as I actually implemented them and figured out more efficient ways to do it, but putting it together got me organizing my thoughts and work flow early on).
With Software I, I was able to breeze through the beginning part and struggled at the end. I had the opposite experience with this project. I was so slow in the beginning I thought I might need the rest of my term to get through this course. But as I chipped away at each requirement, progress began to cascade. At one point I thought I was going to finish by the end of the week and I ended up submitting by that Tuesday evening.
This course is really hefty and worth all 6 CU's! The requirements read as short but there is so much involved in getting through each one. You're also coordinating the database with java and scene builder and at times it felt like a game of pick-up-sticks. For the most part, you can go through the sections by order of the rubric, at least at first. Customers took the longest, almost two weeks. Then I managed to crush through Appointments over a weekend. Reports were done in an evening. Then I spent a few days adding validations and polishing up some of the code.
I didn't want to stop when I did, but there were so many other features I wanted to implement that I figure I'd better save it for when I'm not paying $600 a month for the privilege of staying longer than I needed to. So I have a list of things I want to implement after graduation. I think I did a nifty job for a beginner programmer, almost GitHub worthy. My paper is in the slack google drive. It doesn't give anything away, just a lot of screen shots and names of my files with line numbers when applicable. It's a great example on how to format all papers at WGU (using rubric headers and filling in the blanks).
Here's another slack invite in case you've never made it to the end of any of my posts before now or want to see my paper (join then use the !googledrive command to get a link). Yes I'm there and will respond to messages, though there are a few days during the week in which I only log in long enough to answer messages and catch up on anything I've been tagged in. - https://join.slack.com/t/wguit/signup?x=x-33472218180-427664703120
I have 2 classes left (Introduction to Artificial Intelligence and Capstone).
at No comments:

Wednesday, June 12, 2019

Hired!

After over 14 years at home raising a family, I am officially hired (been mostly on contract or temp positions since December). It is in a great industry (entertainment), the company treats their employees well, the other employees are not insane, the bosses are human, and the benefits are awesome. 
It came with a pay increase so now that I can afford to, I will not be rushing through the end of the degree. That means instead of just learning what I need to finish Software 2, I’ll be working on the Udemy master course that’s been snoozing in my account. I’ll also make capstone something I would want to show off in GitHub.
I've been programming in cycles of: try this, if it works move on, else backtrack to when it was working and try something else. I’m hoping the course breaks me out of that mentality and give me the confidence to work through my bugs or at least figure out what piece of code caused the break.I don't know if I'll use Java for capstone but I need an actual class and I've only heard good things about Buchalka. 
I'm forcing a break because I need to break out of that constant rush mentality. No more worrying about running out of time and money; I have enough of both (now) to get me through another term if needed. 
at 2 comments:

Thursday, June 6, 2019

C950 Data Structures and Algorithms II


I did it! I passed! And I'm not officially qualified as a candidate to Georgia Tech! I have 12 weeks left in my term so the following is just my reddit post. I'll come back to add other thoughts and details when I have more time to myself.  

WARNING: This is long. It was mostly written for those who learned how to program by completing Software I/II at WGU and need a little perspective by the time they reach this point. Remember, I just stared programming a few months ago so if you have a lot of experience, don't expect to be enlightened by my post.

This project tests your ability to take a very large set of constraints and requirements (some of which make very little practical sense), implement them into a design, and code that design to work within the scope of the project. I think I'm finally feeling confident enough to start a portfolio and would if my end of term wasn't approaching so fast. If you are a beginner, I do recommend taking Software II first to get more experience with OOP and putting a multi-component program together. I took this first because it was the last one I needed to qualify for Georgia Tech and I wanted to make sure I got it in before the end of my term but plan could have easily not worked out.

There are some students who built the project way beyond what I considered the scope. There's a lot of interpretation involved, but keep in mind the business aspect and remember they are a small company that delivers 40 packages per day and only have plans to expand in the future. There are things they need now and things they will need for expansion. Are they really going to have the money to invest in extra bells and whistles to accommodate expanding additional cities and whatnot? For a lot of things, it only needs to be designed to allow for future updates (especially for planned expansion), and these are points you'll argue in your paper.

This is the heart of BSCompSci for all CompSci programs. It's a matter of setting priorities and being organized: both in completing the project in general and in executing your solution of the problem and constraints. Break down the problem into some general segments, then break that down further into sub-segments and various tasks (similar to the check list in Software I).

If you are just starting to work with Python, you're going to want to read

Below, I flesh out how I broke down my project to demonstrate how it can be done. Some of my pieces might not make sense depending on how you perceive the problem and how you decide to design your program (the design itself is pretty open-ended which is part of the challenge). I read the scenario as a data problem and reduced it to data that can be stored, retrieved, sorted, calculated, and printed.

That said, if you follow what I did too closely you're going to have a bad time because some of my decisions came about because of how I interpreted the problem, how I coded my project, and the odd twists that lead me to the next part. Remember, I'm still a beginner programmer so there is likely a better way to get it done. I just wanted to write this out for those of you who can't figure out where to start ... break it down and simplify. Remember, your project is going to be built differently so if you use too much of this as a guide, you're going to get stuck as it may not align with your own thought process, flow, or coding techniques. That being said .... here is my breakdown:

Data Structure Stuff:
  1. Build hash table. Zybooks was really good for this part. Pick the one you like, keep in mind you'll need to justify your choice in the paper.
  2. Get csv data into the hash table
  3. Format distance data to suit the data structure you want. You have a few options. A lot of students build some kind of graph. https://www.youtube.com/watch?v=th00zpkPM9Q&list=PLC617CBC8385356FF
  4. Build data structure for packages and distances (the rubric tends to refer to both as package data so it's probably safe to write about both when asked though it probably wouldn't hurt if you picked one and talked about that)
Note: while Python implements dictionaries as hash tables, our assignment is to build a hash table from scratch for the package stuff. I used DictReader for the route stuff.

UI parts:
  1. Look up package data
  2. Insert new package/data
  3. Print all package data at a given time
  4. Print route with mileage data

Algorithm:
There was no breaking anything down for this one except picking one and implementing it. This was one of the times I just had to pull samples apart and put them back together until I could get one running. I found nearest neighbor easiest to understand and model. Dijkstra is really popular because there is an example in the book that makes it easier to implement and I know one person who implemented brute force.

I manually separated the packages into 3 sets (with the help of excel because I'm an excel junkie ... I switched back and forth between sorting by address, delivery constraints, and truck sort to decide what package was going on what route). After I had 3 lists of packages, I ran it through nearest neighbor. Most students opt to program this part into the core algorithm.

... I'm not sure how common it is, but the path determined by my presort and algorithm ended up meeting all requirements (about one of the only things that worked out unexpectedly well!). I saved the things I originally planned to do in terms of optimization for my 'what I'd do if I did it over again' part of the paper. I reasoned that since the company was using csv files, management (or whomever they use to add the data into Excel) can do the initial sort in Excel. I plan to go back to this project after graduation to create a sorting program to replicate what I did in Excel.

Because I worked out the performance of each route did in Excel (delivered time, etc.), I was able to take that experience and translate it into python. The final bits were creating truck objects, having them load the packages and run the algorithm for deliveries. I used those actions to determine the status of packages, start, and end times. This was my outline for figuring out those parts:

Other algorithm tips (not from me):
This info came from a student in chatter who ended up with 76.1 miles. If your presort doesn't match muster or if you think it's too much of a hassle to sort by hand/excel, this was HIS general method: "First part removes all non-deliverable packages. Second part loads all urgent packages. Third part randomly loads packages in several dozen sets. Fourth part recursively checks each set from the third part for lowest mileage, route path, and route weights. Fifth part takes the best data from the fourth part and uses that to cook the route/weights into the truck class. Finally, truck leaves the hub and begins deliveries. Optimization for each part includes loading addresses that share the same address as packages loaded onto truck, unloading non-urgent packages in truck if it exceeds capacity, and throwing out routes if any packages do not get delivered on time."

States, Actions, Limitations:
  1. Truck Limit - 16 packages per load
  2. Package status is either AT_HUB, IN_TRANSIT, or DELIVERED
  3. Truck
  4. Time:
  • Note: I'm pretty sure if truck 2 has 16 packages and leaves after 9: 05 AM, it will not return until after 10:20
  • Every package with a deadline needs to be delivered on time! It helps to have a list for the paper.

For the time component, I kept mapping out what the program was supposed to do and what I needed to get the things that would indicate those things. I started with travel distance and converted that to time. In Excel, I found the difference between the start and end time of each route. From there I was able to figure out how to iterate the time and mileage values for each individual package and during the route (need a cumulative sum to find a package's place in the route). This seems to be a simpler implementation than others who have passed the class (the program acted more like an observer in which events 'happen' linearly as in the 'other algorithm tips' above). I'll chalk off my interpretation to my beginner status. The jury is still out on whether my method is scalable for calculating delivery times and setting the status.

Again, try not to tackle more than one issue or component at a time. If you're getting overwhelmed, you're probably trying to do too much at one time. Take a step back and figure out what you need first, then limit yourself to working with that. Break it down until the only parts left are simple.

The Writing Part:
The writing component is incredibly ambiguous; you need to combine the instructions with the rubric guidelines to make sense of it. I pasted the instructions into a document and used the various sections as topic headers. Then under each header, I pasted the rubric stuff (in italics). I highlighted them a really ugly color so when I was working on it, I could easily tell if I were done with the section or not (removed highlighting when done).

Some of the sections are basically asking for the same thing; it's ok to copy something from one section and paste it into another. For example, there are two places that ask you to verify that each package was delivered on time. Yes, they want it again; no, you don't have to reword all the iterations of the same question. Some sections ask for things that aren't included in the scope of the project; for those I said: the current software requirements don't include ___. For example, there was one section that mentioned hosting of the environment. For brownies, I followed up with: implementation of that feature would require ___. It was helpful asking about sections on Slack and discussing how everyone else interpreted the question and how they answered it. There were variations and all were accepted.

Anyway, I was nervous about the complexity analysis because I wasn't sure how well I understood it (I would not be surprised if I got some of it wrong). I took my best guess for each function in each script (in a note), then created tables for each script in my paper where I listed the line #, space complexity, and time complexity. At the bottom of the table I added each line, then put it in Big O notation. Here's a short example:

buildhash.py
Line #
Space Complexity
Time Complexity
11
O(N)
O(N)
20
O(1)
O(1)
38
O(N)
O(N)
52
O(1)
O(1)
Total:
2N + 2 = O(N)
2N + 2 = O(N)

Once I did that for each script, I replaced line # with Script Name and listed the results. Then I added to get the total N's and converted it back to Big O notation ... I didn't go into that much detail for the core algorithm overview. For that one I just put the overall Time and Space complexity in a section of my overview which is in the sample the mentors give us. I think they would notice if you had a sort function and the complexity didn't include a log function and things of that sort so I wouldn't attempt to put in random values. Do your best.

Anyway, I'm down to 3 courses: Software II, Introduction to Artificial Intelligence, and Capstone. Woot!

at No comments:
Subscribe to: Posts (Atom)