Today's research relies on trustworthy software. Many scientists develop their own software; however, the quality of academia-developed software tends to be lower than commercially-developed software because, in academia, there are barriers to using proven software engineering methods. To help overcome these barriers, the Water Science Software Institute (WSSI) has developed a model--the Open Community Engagement Process (OCEP)--which brings software engineers and scientists together to traverse a four-step, iterative process that incorporates Agile development principles and open source mechanics. As a part of OCEP, WSSI has engaged a water science community who uses a scientist-developed, computational modeling framework originally developed in the early 1990s. Efforts to improve this software have included two hackathons. This paper compares these hackathons to determine factors that influnenced hackathon outcomes. Thorough planning with sufficient lead time before a hackathon, clarification of expectations, sufficient time for discussion of objectives and clarification of domain vocabulary, and co-location of participants were identified as key factors contributing to hackathon success.