CC14 - Prototype search application to index climate-change and environmental websites using structured metadata, and display the results

Project start and end dates:
2020-01-18 to 2020-05-02

Goals & Objectives

Build a prototype website/search engine to help us index climate change and environmental websites and company information for a new climate change impact software product we are designing.

Create a simple prototype to demonstrate what is possible.

Connect to the Web Data Commons data set (if the data is too large, we can setup an AWS environment for you in our account). Then using example keywords we provide, look relevant metadata and extract those results to a web page that indexes them by type (ex. web page, article, organization, event). Store the results in a database and/or elasticsearch index. Create the ability to query the results and export as CSV, or with the original structured metadata.

Project Outcome
We are performing a lot of market research on climate change and are expending a great deal of manual effort to identify relevant websites and organizations. At the same time, many websites now use rich microformats and semantic metadata to describe their articles, events, and company information. This information is captured in the Web Data Commons project ( which uses Common Crawl ( The ASU students were tasked with developing a prototype search engine to extract from the Web Data Commons a list of relevant websites, articles, events, and organizations, and display those in a search index where we can view, search, and export it. While the project work was complex and only intended to be an early prototype, the team produced results of professional quality and complexity, leveraging Amazon Web Services infrastructure to create a scalable and performant big data indexing solution. They demonstrated enthusiasm, architecture and development skills, creativity and insight throughout the project. They were always pleasant, professional, keen to understand our project goals and objectives, and willing to go the extra mile to help achieve them.


We would like to thank Pradeep AJ, Yuvan Pradeep, Ankita Shivanand Bhandari, Harika Kolli, Narendra Mohan M, and the other students and teaching staff of Master’s Software Engineering Capstone course. *

* For privacy reasons, we only list people who gave us permission to do so. Did you contribute to this project? Contact us to be added!

Related Project

Climate Change Impact Planner
When it comes to climate change adaptation and mitigation, we believe there is a critical communication gap between government officials, scientists, and experts (“trusted authorities”), and ordinary citizens. As citizens, we do not know what specific actions we can take to avoid, mitigate, adapt to, or recover from the effects of climate change in our personal situation (such as in our home and neighborhood). Even if we do act, trusted authorities do not know if we are following their guidance and cannot study the effectiveness of our actions. The anticipated product will use satellite Earth Observations, big data, and machine learning to identify past climate change impacts and predict future impacts. It will focus on end users and stakeholders (such as municipal governments or insurance companies) who need to understand and plan around climate change impacts without requiring them to know the underlying science or technology.

Academic Institution

Arizona State University

Tempe, Arizona

Arizona State University is a public metropolitan research university on five campuses across the Phoenix metropolitan area, and four regional learning centers throughout Arizona. ASU is one of the largest public universities by enrollment in the U.S.

A Riipen Project

Riipen is your online platform for virtual project-based learning

Get hands-on support from our students through an in-class project or virtual internship.

  • In-class projects allow you to connect with one of our educators to embed your project into the students’ curriculum. Become the real-life case study for students in the classroom!
  • Virtual internships are similar to in-person internships, except they are project-based with a clear outcome and the engagement is primarily done online.