The week started off with me doing the same thing I did last week, which was porting code from Java to Groovy, and running the daily data build. After becoming familiar with the build, Chip revealed to me (and the rest of the data team) why he had me join the team – while the other team members all worked on keeping the build stable and making incremental improvements, he wanted me to implement a more centralized and accessible logging system for the entire build, so that in the future, keeping tabs on the build and knowing the necessary time  and order for the execution would be a lot easier to manage.

This was exciting – a chance to put something into the build that would last multiple years, and actually be useful and increase the productivity of those who interacted with the build – I was thrilled to work on this. Chip had already written a program that helped append the log messages to the messaging app that StockCharts uses, HipChat. Since the HipChat appender was already created, all I had to do initially was connect the current java part of the build to the program. This turned out to be a lot simpler than expected, and I soon finished, pushing my code to production. (By the way, pushing code that you know is going to be in production for the fist time is an intense feeling, much akin to the feeling you get when you’re expecting a reply from someone and are just about to open their message.)

I didn’t break the build! Next, it was time to move on to groovy, which should pose more challenges. However, just as I was about to start, I was warned by some other members of the team not to make any changes to the groovy build temporarily – they were working on a certain feature, and they didn’t want any version control nightmares, or any broken code. So, as I sat around, I decided to try and automate the daily build. I was having to run the daily build everyday while finishing my coding assignments as well, and it was a tedious task. I reckoned that with a little touch of crontab and the right commands pushed through, the daily build wouldn’t need people to run it ( except the parts that needed to be reviewed, of course). I sat down and researched cron jobs, and learned a lot about cron and the unix shell in the process – the amount we can do with our computers is amazing!

That’s pretty much all for this week though. Will I finally start writing some groovy logging? Will I be able to automate the build in the manner I want to? Find out in the next episode of Rajesh’s Summer Adventure!

Week 8 – Running the data build

Leave a reply

We finished the quiz application! Well, almost. Having finished creating all the functional requirements of the quiz, all that remains is for us to put the final touches on the application, then connect it with the final website so that it goes live. It’s pretty exciting to see something we’ve been working on for so long finally come to fruition, but at the same time, we’re all also pretty happy to be done with working on this project. It’ll be interesting for all of us to move on from this project, and see where the rest of the internship takes us.

On that note, since the final touches that are to be made are mostly to do with the front end aspects of the application, Quinn and I don’t really have much to do. As such, Quinn was given another project to work on, and I was assigned to work with the data team! I work with QA, another Whitman alum, on the data builds and the daily data processes. Most of our work is making sure there are no anomalies in the code and the output, and so we execute production code and make sure everything is up to the mark. On the side, however, we work on porting existing groovy code over to java, which is both fun and confusing – groovy is similar enough to java that you think you would be able to do it without much thinking, but different enough that you run into all kinds of obstacles when trying to make something work in java. A lot of the libraries used in groovy are slightly different from their java counterparts, which makes it so that we often have to come up with new solutions to make something work, as the libraries used in the existing codebase are different from the ones that are available to us.

That’s been all this week, it’s become a routine of running the existing code in groovy and porting it over to java. Stay tuned to find out what happens this coming week!

Week 7 – FileUploader

Leave a reply

I spent some time last week trying to create a program that would allow users to upload different types of files using Amazon’s DynamoDB, so that it could be used later to create quiz banks. However, there was another part that I had to complete, which was to do the same thing, but instead to upload the files to S3 so they would be accessible from the main program, and in the form that we want.

This week was shorter than the others, so we had only 3 days to finish our weekly targets. I spent all three days trying to figure out how S3 worked – you would think that Amazon makes their technologies similar to each other, and therefore easy to learn when you already know how to use a different one, but no, they’re all different. I learned this the hard way when I tried running the entire program we had up to that point along with what I thought the solution should be – it gave me enough red signals to send me running back to do a better job.

I proceeded to try all kinds of solutions – tried uploading the files directly, changing the client I was using to upload the files to S3 (you wouldn’t think this should have been done, but due to technologies that were later deprecated, this turned out to be surprisingly helpful) , and even changing how the files were being uploaded – I tried them all. The point where I hit rock bottom, I think, was when I spent 2 hours trying to figure out how to convert a temporarily uploaded file to a byte stream using Amazon’s input API (Amazon is very finicky about the types of data you can upload and then have be accessible later), then figuring out it’s location and giving it full file status while trying to upload it to the database – it took a turn for the worse, and I was not a happy camper. However, I seemed to be on the right track as far as converting the input to a bytestream went – from there, it just became a matter of changing the file type of the recently uploaded file from a single part to a multipart, and remove all encoding so that the doPost method of the program would be able to understand the file that was being sent its way. Finally, I was able to accomplish what should have been nothing more than a 2-3 hour task!

Nope. There was still the matter of the client, and the HTML form through which the file was to be uploaded. I powered through the rest of what was to be done, and finished it barely 15 minutes before the last bus was to leave work – then ran downstairs, and made it onto the bus with seconds to spare.

This week was good, as I learned that programs never really get done in the timeframe you think they will – some don’t take very long, and some, like the one I tackled this week, take you a seriously long time.