CC14 - Prototype search application to index climate-change and environmental websites using structured metadata, and display the results

Project start and end dates:
2020-01-18 to 2020-05-02

Goals & Objectives

Build a prototype website/search engine to help us index climate change and environmental websites and company information for a new climate change impact software product we are designing.

Create a simple prototype to demonstrate what is possible.

Connect to the Web Data Commons data set (if the data is too large, we can setup an AWS environment for you in our account). Then using example keywords we provide, look relevant metadata and extract those results to a web page that indexes them by type (ex. web page, article, organization, event). Store the results in a database and/or elasticsearch index. Create the ability to query the results and export as CSV, or with the original structured metadata.

Project Outcome
We are performing a lot of market research on climate change and are expending a great deal of manual effort to identify relevant websites and organizations. At the same time, many websites now use rich microformats and semantic metadata to describe their articles, events, and company information. This information is captured in the Web Data Commons project ( which uses Common Crawl ( The ASU students were tasked with developing a prototype search engine to extract from the Web Data Commons a list of relevant websites, articles, events, and organizations, and display those in a search index where we can view, search, and export it. While the project work was complex and only intended to be an early prototype, the team produced results of professional quality and complexity, leveraging Amazon Web Services infrastructure to create a scalable and performant big data indexing solution. They demonstrated enthusiasm, architecture and development skills, creativity and insight throughout the project. They were always pleasant, professional, keen to understand our project goals and objectives, and willing to go the extra mile to help achieve them.


We would like to thank Pradeep AJ, Yuvan Pradeep, Ankita Shivanand Bhandari, Harika Kolli, Narendra Mohan M, and the other students and teaching staff of Master’s Software Engineering Capstone course. *

* For privacy reasons, we only list people who gave us permission to do so. Did you contribute to this project? Contact us to be added!

Related Project

OASIS: Software Solutions for Climate Change Problems

Software can help the world respond to climate change impacts and disasters.

In 2019 we set out to investigate if there is a need for a solution that gives citizens personalized help preparing for and recovering from climate change impacts and lets trusted authorities assist them and report on their results.

As part of this effort, a great deal of market and technical research was conducted over the last few years. Over 140 students have been involved so far, in dozens of academic institutions in Canada and internationally.

Our greatest finding is the willingness of citizens to take responsibility and help themselves, their neighbors and community plan for, mitigate and recover from climate change impacts and disasters. We also confirmed that a software solution can be built to help overcome the barriers they face when trying to do so.

Now we’re working on a prototype and sharing our findings and progress. Find out more on our OASIS project website!

Academic Institution

Arizona State University

Tempe, Arizona

Arizona State University is a public metropolitan research university on five campuses across the Phoenix metropolitan area, and four regional learning centers throughout Arizona. ASU is one of the largest public universities by enrollment in the U.S.

A Riipen Project

Riipen is your online platform for virtual project-based learning

Get hands-on support from our students through an in-class project or virtual internship.

  • In-class projects allow you to connect with one of our educators to embed your project into the students’ curriculum. Become the real-life case study for students in the classroom!
  • Virtual internships are similar to in-person internships, except they are project-based with a clear outcome and the engagement is primarily done online.