Please someone tell me how to write generic code for specifc tasks

Heyo,  Joe here, coming off of a great semester of working with Marcus to coerce our data to play nice with Python.  Our goal was to unite the geographic data we had for the area upstream of each point with the isotope data from Harbin, our hard working germanium detector, into a format that could be sensibly manipulated with Python for graphing and statistical purposes. The problem was twofold, figuring out a way to sensibly store and access data, and how to take that data and use it to make graphs that were understandable and looked nice. I dove into writing a whole tangle of functions to pull out the data of interest and Marcus became good friends with the matplotlib documentation, his only ally in the noble fight against the matplotlib library.

It must’ve been just about two years ago now that I first started to truly get my hands dirty with both Python and ArcPy, ArcGIS’s Python library.  I started with a simple goal, create unique watershed files for each point in a shapefile full of sample collection locations.  Through a combination of the ArcPy documentation, stackoverflow answers, and a dear friend of mine with far more Python experience than myself, I was able to create such a script.  It was tailored to my specific project, but I tried my best to make it something that could be reused for other projects.  Looking back now, I would do it all totally differently, but ya live and learn!

When I began work on my next script, which extracted spatial information for each watershed, I became consumed with finding my way around ArcGIS’s ‘table joins’ which is perhaps the most obtuse way to unite two sets of data.  I won't go into detail, but I accomplished my goal, learning a lot about how ArcGIS stores data in the process, and thus began the quest that still consumes me this day, which is to avoid using ArcGIS at all costs, offloading as much work as possible to Python.

In the fall, I declared, to no one in particular, my intent to secede from ArcGIS, and began work on a Python project to manage my data, which would only dirty its feet by dipping into ArcGIS as needed for certain spatial analyses, then whisking the results out of the clutches of whatever heinous file that Arc would create, and into the sanctuary of my Python datatype.  Progress was slow, mainly because I kept on trying to start over!  My code worked fine, but I was never satisfied with how it was structured, I wanted this to be something that people doing similar, but distinct, work could use.  I struggled with how to avoid design decisions specific to my project, which was hard to do when I was also trying to use it at the same time to do my actual project. Eventually looming deadlines (apparently you need “results” when you “present” at a “conference”) forced me to move forward, so I ended the semester with a datatype to store the data about my samples, some functions to grab that data, and some functions to graph it.

Now, from what you read above when you saw “functions to graph” you may have thought to yourself, “oh, this must be where matplotlib comes into play” and you would be right, if I had an ounce of sense in me.  For a reason I am unable to explain, I’m not sure if it was ignorance about the existence of graphing specific libraries like matplotlib, hubris, or just naive fondness for LaTeX, I decided to write functions to generate the markup to generate plots using the PGFPlots package for Latex.  This meant that instead of calling functions like plt.plot(), I was writing long format strings to generate a file in the LaTeX markup language.  The results were rather pleasing, but when I came back to the project in February with Marcus I thought a more straightforward approach would be appropriate.  When we found out about matplotlib, I thought, “Now here is the answer to all of our problems!  All we have to do is hook up the Python code that stores the data to matplotlib and out will come beautiful graphs”.  Sure, I thought we might have to do some tweaking to get graphs up to our very refined standards, but how hard could it be.  For that answer, see Marcus’s post.

The template...

...and the result!

So, as Marcus went off to figure out just how to make matplotlib give us graphs that could be read with ease, I went off to figure out how I could pull the data we wanted out of the jumble of samples we were working with.  Our dataset was a collection of 83 soil samples from three different field seasons.  For each sampling location we determined the area upstream of it, and calculated various geographic parameters.  

Now that I had gotten all this data, it was time to get organized logically.  This took some time, but boy was it worth it.  Once I knew that all the data would be have the same way, I wrote a series of functions (way too many functions, probably, but once you learn Lisp, there’s no going back) to return lists of the data we actually wanted to plot.  If each sample has activity and error values for 3 different isotopes, a thousand different geographic parameters, a location, links to files, lists of other samples that this sample is related, it’s not quite plug-and-play.  But it got done, and meant that if some of the values for our samples changed (as they often do) or if new samples got added or old samples got removed, as long as they conformed to the standards, we didn’t have to do a darned thing!  Just take the list of sample objects, plug it into the function that pulls out the data you want to graph, and then shoot the result of that into Marcus’s graphing code.  Badadbing badaboom!  

    Things should be smoother from here on out...until I finally figure out that perfect structure and write the definitive program for managing soil samples, computing watersheds, doing some spatial analyses, and plotting and tabulating the data.  Someday it will happen, and we will be better for it. 

It's been a great 2+ years working for OGRe, but I wouldn't be surprised if I come back in one form or another, even if it's just to preach about why we should be scripting more and clicking less.

Lab Work (Spring 2016)

Hi all, 

Marcus here, and ready to share with you what i've accomplished in the while working in the lab this Spring. This project was quite an undertaking from the very beginning; I had just conquered the mighty task of “Hello World” in java when I found out that I’d be paired with Joe to prevent more Excel made graphs from entering publication. We were given the option to either work in R, a completely unfamiliar language to both Joe and I, or to see if there were any ways to get Python to co-operate. Enter matplotlib (MPL).

This library was so combative that we had to make use of a separate application called Jupyter, which already had MPL integrated in it, to start working. Determining a starting location was a task in itself. Our first graph made extensive use of an oh-too-kind stackoverflow users code which gave us a 7x7 grid of information. That was a lot to take in, so we started looking into how get more specific with what we were presenting. The next graphs we created were specifically targeting lead, in-channel vs overbank and resample vs original, including variants that had data that accounted for negative values and those that didn’t. The nature of working with lead values is that they all had crazy error-bars, which got to be distracting visually. Suffice to say, there were a lot of moving pieces that didn’t want to work together at first.

This is when we started changing up our approach, Joe had delegated me to be more in charge of creating the graphs themselves while she continued working on her already built code that was able to grab and pair related information sets. My job required me to understand what data I was being given and how to use that in the graphs I would be creating, so naturally I needed to understand, at least some parts of, the code Joe was already working on. Now my to-do list included: learning how Jupyter worked, learning how to use MatPlotLib, understanding how Joes code was structured. Maybe it would’ve been easier to work in R. Jokes aside, it was a somewhat daunting task, so I figured it’d be best to start by seeing how Joes code worked, as she was much more accessible than the authors of the other two applications.  

It was a really interesting experience getting to see how Joe went about setting up her code to retrieve data and return meaningful results. The structure she had set-up worked in a way that was fairly straightforward, so figuring out how to add to and build off of what was there already wasn’t too difficult. It was a nice introductory period before we got our hands dirty with MatPlotLib. As I mentioned earlier, we really started off making plots that looked nothing like what our final graphs became. It was definitely a learning experiencing as we figured out what worked best and how to get information dense graphs that were still intelligible. Stackover-flow and the many documentation web pages quickly became purple links in my google searches as to why this wasn’t working or how to change this seemingly obvious part of the graph. Changing from regular plots to scatterplots, playing with subplots and legends and even changing the font size, color and shapes all seemed to have specific intricacies that want to provide un-intuitive results for seemingly small modifications. Ultimately though, Joe and I were able to work through these bugs as they popped up and create some good figures in the process.

This semester was a great introduction to working in a lab and I'm exciting to see what the next several years will bring!

STEM Night

On Friday 6 May, the Geomorphology Group joined other geology students to participate in STEM night, an outreach event for 3rd-5th graders. We had a great time. Below are pictures of OC students interacting with elementary school students. They are playing with the stream table (with Marcus and Adrian), exploring mineral properties (with Andrea), and looking at fossils (with Alex). Sydney and Andrew aren't pictured.

Winter Term and Beyond!

Hi everybody! I hope you all had a wonderful Winter Term and are ready for a semester of research!

As stated in my previous blog post, I spent the Winter Term working on the SEM in order to determine whether it could be used to achieve quantitative elemental information. By comparing my data from the SEM to the data sent over from the College of Wooster by Meagan, who analyzed the same samples using an XRF device, I was able to determine which factors on the SEM to set in order to achieve the closest results.  In short, I determined which factors (such as magnification of the sample and how much time is spent analyzing the sample) are the best for achieving as close results as possible compared to the results achieved by Meagan using her XRF device. I spent the last week of Winter Term learning how accurate and precise I could get my data and also spent some time learning how to incorporate new standards into the SEM for future analysis of other samples.

            The upcoming semester will be spent continuing this research. I will determine a protocol to use the samples sent from Wooster as standards for analyzing ground up sediment by using fractions of the samples sent over by Meagan to try to further replicate the results by using different sample preparation techniques, such as dipping SEM tape in sediment, making grain mounts from the powdered versions of the samples, and even making dirt thin sections (hopefully with the new thin section machine we got in January!). In the future, I also hope to compare my results to samples that were analyzed separately at the University of Vermont using a portable XRF device. I will also eventually be using other mineral standards with the same protocols on the SEM.

            Adrian and I will be working together on a research project for the upcoming semester. I spent Winter Term preparing for this by reading through various papers related to radionuclides and their effects on erosion and the interpretation of particle size effects on the adsorption of certain isotopes, such as ¹³⁷Cs and ²¹⁰Pb.

            I will talk further about my results and my future plans at the lab meeting this coming week. I look forward to hearing about what other people did over Winter Term!

-Chloe

Winter Term 2016: Marcus' Post

Hello everybody! The bulk of my first Winter Term at Oberlin was spent doing research in China with a crew of others who all seemed to be geologists at heart. This trip was eventful from the get-go, from dealing with crazy TSA lines and unfortunate weather delays, to overcoming the local cuisine, there was so much more of everything than I had anticipated.

We spent the first two days in Chengdu while waiting for my delayed luggage to arrive in China. This free time allowed us to have a little bit of the touristy experience, so Maisy and Liam took Al, Zanna and I to see pandas at the Chengdu Panda Research Base. After spending a few hours there, we all crammed into one taxi and made our way back to the university to explore for a bit. This was followed by listening to a talk about the relationship between health and air quality within China at Sichuan University. Not surprisingly, there was a correlation between increased pollution and health hazards, but what was interesting was that transportation output didn’t seem to be the biggest factor. The first day was really enjoyable as we were able to see firsthand things that were distinctive and relevant to China, like pandas and unfortunately pollution.

As for the actual field work, we sampled from a variety of sites, each unique and holding their own entry hazards. With the first couple it was mainly figuring out a way to navigate through the garbage. The first in particular wasn’t too bad as it was at the base of a construction site, while the next few really involved a lot of maneuvering around decent sized cobbles. Getting down was only half the battle as finding a good spot along the river that had sandy material to sieve, rather than mud, wasn’t easy. Although having so many people willingly to get their hands and feet wet really made the process easier.

Speaking of tasks that weren’t easy, Amanda really had the worst of it with some of the driving she had to do. Both she and the rental car deserve at least a few medals for going through what seemed like worse and worse road conditions every day. As we were heading up a mountain one night a blanket of fog completely engulfed the car, reducing visibility to maybe 5 feet at most. There were also the countless wēi qiáos (dangerous bridges) that were both crossed and thankfully just passed. Tunnels that stretched on for kilometers, without and without lights, and of course other drivers. As tense as some of that driving was, it really did provide for some of the best views you could get while traveling. There were several times where it felt incredibly surreal to be looking out and see such impressive terrain right outside the window. Coming back home to the mid-west where the land is all flat, I definitely miss having such amazing landscapes before me on long car rides.

Getting to try all sorts of different food was also a fun time. As someone who isn’t particularly skilled in using chop-sticks it was a task just getting it from the serving plate to my bowl, but by the end I had improved considerably. Although I am an adventurous eater it seems I’ve got a fairly weak stomach, and even trying to stick with the less spicy stuff, authentic Chinese food got the best of me. Nonetheless this trip was an amazing adventure to go on, missing luggage and stomach bugs and all. As Zanna mentioned, exploring China for its geologic information provides an experience that is completely unlike one you’d get as a tourist or someone on business. Getting to spend just over two weeks in great weather doing field work for the first time really made this a fantastic way to start the New Year, and im excited to see the data that will come out of it. I’m so grateful that I was able to be a part of this trip, and really hope to go on more in the coming years.

Marcus

Left to right: Amanda, Maisy, Liam, Marcus, Zanna, Al, Wong Yu Mei

Winter Term in China: Zanna's Take

Hey everyone! This winter term I was in China with Amanda and Marcus. I flew out to Chengdu on Jan. 4^th and was there until the 20^th collecting sand for ¹⁰Be testing in order to determine upstream erosion rates along the Jinsha River. Starting out there was actually 7 of us since it was Marcus and me, Al, Wong Yu Mei, and Liam and Maisy. It was really fun to be in such a big group that was so excited to be doing fieldwork. It didn’t hurt either that three people in the group were fluent in Mandarin so getting around became relatively easy.

After a day in Chengdu, we started traveling south to our sample sites. We did a whole lot of mountain driving and off-roading through rural villages in addition to lots of climbing down slopes off the sides of the roads with our shovel, sieves, and sample bags. Twice we ended up in traffic jams on the side of the mountain because they were still building the road in front of us! Most of the time we found partial roads down to the river, including our second site, trash kingdom. The site was below the highway and there was so much trash collecting in part of the river we all initially assumed it was a sandbar. So despite the fact that the air was pretty clear, it wasn’t always a great thought to think about how much polluted water we were sieving. We had two other big adventure days: once, we hit the end of the road where the bridge across the river was still being built. Amanda hired a boat and we climbed down under the bridge to take the boat to our sample site. It was cool to see the landscape from a boat in the middle of the river, rather than from the side. Bonus: our boat driver was the 10^th greatest hero in Yunnan; apparently you can look him up on Baidu (China’s Google equivalent). The other day Marcus missed due to sickness. After driving up into the mountains we took a hairpin road on the edge of a cliff that had the mirage effect of looking better than it was, but was all landslide rock. As a testament to how “in the middle of nowhere” we were, halfway down we came upon a guy and his donkeys who looked shocked to see us and answered with, “You’ve made it this far” when Amanda asked if the road led to the river.

We traveled as far south as Panzhihua, where it was sunny and in the 70s every day. It was really nice to be sieving in the river then. It also seemed each of our sites was prettier than the next! Throughout the entire trip I enjoyed the food too. We did a lot of veggies and rice, but also tried fish a couple times and of course had noodles and steamed buns. The yogurt was probably my favorite snack along with their milk tea. We finished collecting our samples early and managed to do a few cultural things upon returning to Chengdu. We went to an archaeological museum, the coffee district, and the walking streets; it was such a contrast to the poorer communities and minorities we’d seen in the mountains, but good to see both sides. Overall, it was a great trip and interesting to travel in a country based on geology rather than tourism or business. Not to mention, I learned a lot about China in addition to gaining experience with fieldwork! I’m excited to see/hear how the analysis side goes when our samples eventually arrive.

Winter Term with the SEM!

Hi everybody! Chloe here. I’ve been spending my Winter Term at Oberlin doing various things, mainly working with the SEM down on the second floor of Carnegie! Amanda gave a brief introduction on my project during one of our lab meetings in the Rat, but here is a reminder: I’m spending the Winter Term getting to know the SEM and using it to develop a method for giving us quantitative elemental information for materials, such as finding out the bulk compositions of different samples of sediment. I am using samples sent over from the College of Wooster by Meagan Pollack, who did analyses of her own before sending them to me. 

The past two weeks have been a LOT of fun and I’ve learned tons of new things, ranging from how to work the SEM and the theory behind it, to learning how to run sediment samples through HARBIN and even filling the liquid nitrogen detectors in Amanda’s lab and in the SEM. The first week of Winter Term was spent getting to know the Scanning Electron Microscope by taking advantage of the copious information available through the “help” system installed in the Oxford INCA software. I also spent some time reading through a helpful paper that Zeb gave me, titled “Performing elemental microanalysis with high accuracy and high precision by scanning electron microscopy/silicon drift detector energy-dispersive X-ray spectrometry (SEM/SDD-EDS)” by Dale E. Newbury and Nicholas W. M. Ritchie. Zeb also loaned me one of his books, “Modern Analytical Geochemistry: An Introduction to Quantitative Chemical Analysis for Earth, Environmental and Materials Scientists,” which provided further useful information concerning elemental analysis through the use of spectrometry. I spent the rest of the first week getting used to the SEM through hands-on work in order to re-familiarize myself with the machine. I had not used the machine since a couple of weeks before first semester ended when I used it for the Mineral Drawer Project in Mineralogy.

On a side note, for those of you who remember using the SEM in Mineralogy, I am using the machine in “Analyzer” mode instead of “Point & ID” mode. The “Analyzer” mode is different because it can analyze a bigger area than the “Point & ID” mode. It works in a slightly different way, but is just as fun as the “Point and ID” mode!

            The past week was spent collecting data on the SEM using the samples sent over from Wooster by Meagan. I also learned how to prepare and mount these samples, which includes learning how to use the carbon coater machine in the SEM room (which coats the top of the sample in carbon) and further preparing the samples for use in the SEM by putting them on a metal holder and attaching small amounts of copper tape to the sample and the metal holder in order for it to be conductive. I also spent the past week conducting beam stability tests with the SEM, which included taking analyses of the copper tape that is attached to my sample in the SEM and doing quantitative calculations to see how stable the beam current (of electrons that hit the sample) is over a period of time. I’ve learned that it is best to let the machine “warm up” for an hour or two before performing more in-depth analyses on my samples.

            I also spent some time over the past week learning how to make thin sections (with varying degrees of success) with Adrian. This past weekend was spent going over my data and trying to replicate the analyses that Meagan made at Wooster using the same samples. Next week I will conduct more stability tests with the SEM (except by utilizing different timing for the tests, such as leaving the machine on for longer and then doing a quantitative calculation), and work on determining a protocol to use the samples from Wooster as standards for analyzing ground up sediment. Adrian and I have also spent some time going over what we will do for our research project next semester! I’m coming into it with beginner’s knowledge, so I’ve spent a lot of time reading up on papers explaining isotopes such as ¹³⁷Cs and ²¹⁰Pb and their effects on soil erosion and the interpretation of particle size effects on the adsorption of certain isotopes.

            I’ll just sign off here and make another post next week. I hope everybody’s Winter Term projects are going well and everybody in China is having a great time! J

-Chloe