Investigators' Blog

The Archaeology and Physics Departments at the University of Auckland, as well as contributors from other universities, have been collecting data on obsidian artefacts from the north part of New Zealand. To date, this project has data on over 2,500 such artefacts, obtained from various sources including historical studies done on obsidian to more recent studies done by current archaeologists at the University of Auckland. Part of the aim of this research is to look at “Social Network Analysis of Obsidian Artefacts and Māori Interaction in Northern Aotearoa New Zealand” which is the title of a recent publication which involved my Te Pūnaha Matatini and industry supervisors.

Why study obsidian?

Obsidian is a volcanic glass which is found at several locations in New Zealand. It is hard and brittle such that when a piece is broken off (called a flake), it has sharp edges. This made it very useful as a cutting tool in pre-European New Zealand. By analysing the elemental compounds of the artefacts, it can be determined where each artefact was sourced. By comparing this to which archaeological site each artefact was found at, my supervisor Dr Dion O’Neale has been able to infer social networks of pre-European New Zealand. Dion analysed the geographical least cost paths and found that distance was not always the main factor in determining where each archaeological site sourced its obsidian flakes from. Therefore, by analysing obsidian artefacts, a lot of information can be gained and it is the aim of this research project to be able to infer this type of information and even more regarding pre-European Aotearoa New Zealand.

With so much varied data the need arose to have a central data infrastructure where all the various data records can be stored along with protocols to support data quality and provenance. This data needed to be accessible by various parties from various departments and universities.

The main steps I took to complete my internship project included:

• Choosing and learning to use an appropriate database software
• Schema design
• Data cleaning
• Scripting for automated data uploading

These steps were not necessarily sequential and often ran in conjunction with each other. For example, since there was a variety of data sources, while I was doing the data cleaning I came across new data fields in which case I had to edit the schema to reflect the new field. However while doing the data cleaning, I often came across discrepancies or unknown variables in the data which I needed to wait to hear back from other people about before I could proceed.

It surprised me how long it took me to design the schema. Data cleaning often takes the longest amount of time. In some sense the data cleaning did take some time because while I was designing the schema, I was also figuring out what data to keep and what not to keep. This greatly reduced the time it took for me to clean and format all the data tables to be ready for upload. After that, finding and learning to use an appropriate database platform also took a while. Finally, writing the scripts for automatic uploading to the database took a couple of weeks.

Author

Kate is currently studying for a Master of Applied Data Science (MADS) at the University of Canterbury. 

For 10 weeks over the 2018-19 summer, I was involved in a project with Te Hiku Media and Dragonfly Data Science to aid in the development of a Māori voice assistant. The motivation for this project was to make Te Reo Māori more accessible and fun in the digital age.

During my internship I achieved the creation of a “box” called Rapere (translation of “Raspberry” into Te Reo Māori) containing a Raspberry Pi computer which is connected to the internet, some lights, a speaker and a microphone. This box has been coded to be continuously listening for spoken voice, and when this is detected it records what is being said until there is a longer break in the speaking (this file is overwritten each time a recording is made).

Cherie’s recording equipment set-up.

The recording is transcribed using Te Hiku Media’s Application Program Interface on koreromaori.io. The transcription that is returned to the box is compared to some key words which mean the speaker is likely asking to hear the news or to listen to the radio, or to stop playing. If these are heard, then the news or radio stream is played or stopped, and otherwise it goes back to listening for these phrases. The phrase “kia ora” lights up an LED for a few seconds. The box is able to listen for commands while playing audio, which allows the user to stop audio playing. The project was documented and all the code uploaded online to allow other developers at Te Hiku Media to progress it further and demonstrate the abilities of the Rapere box.

I experienced a great feeling of accomplishment from my work with Te Hiku Media and Dragonfly Data Science. Going from a bunch of components and an empty raspberry pi computer to having a working program with two different APIs and which plays the news on my correctly saying the appropriate phrase in Te Reo was more than I thought I would be able to achieve and I am proud of what I achieved with the help of my supervisors. I am grateful to Te Pūnaha Matatini for providing me with the opportunity to have this internship.

Author

Cherie Vasta is a student at the University of Canterbury who is going into her final year of a Bachelor of Engineering in Mechatronics. Cherie enjoys problem solving and working as a part of a team. She is excited by working with high-end technologies, as she would like to be at the forefront of engineering the future.

Romalee Amolic talks about her 2018-19 Te Pūnaha Matatini Summer Internship with AgResearch where she worked on a project to enhance social network analyses of biosecurity information in the New Zealand tourism industry, so that such analyses can be conducted faster and more effectively in the future.

Social Network Analysis (SNA) is a powerful data analysis technique which often helps in identifying hidden relationships and other critical information in a communication network.  Data for an SNA can be collected from various sources which may result in extensive pre-processing and cleaning time as compared to the time needed for actual network analysis. Hence, this project aimed to use data carpentry to streamline the use of social research data (e.g. collected through surveys) to be able to conduct social network analyses quicker and more effectively in the future.

Better Border Biosecurity case study

The Better Border Biosecurity project, a multi-partner, cooperative science collaboration which analyses the exchange of biosecurity information in the New Zealand tourism industry, is used as a case study to develop and test methods which streamline the SNA process. The data for this project includes 154 responses from tourism providers across New Zealand who named up to 3 sources from whom they seek or receive biosecurity information. Information about the location and role of the respondents was collected. Some additional questions were also included in the survey such as the form and frequency of communication, the usefulness of information and the trust between parties.

This information was then used to perform Social Network Analysis in Gephi – a powerful interactive social network analysis tool. However, the survey data had to be first converted into a format fit for network analysis. The conventional approach for cleaning the data is discussed below.

Conventional approach

The survey responses were collected in Excel sheets. The data pre-processing and cleaning was done manually using Excel.

Problems with the conventional approach

• It involved dealing with the data manually which was an extremely time-consuming process and needed about 1-3 weeks depending on the complexity of the data.
• The process was prone to human errors which reduced the potential of the data.
• It required skilled labour for an extended amount of time and hence, increased the costs involved.
• It led to data inconsistencies.

Hence, taking all these problems into consideration, a generic automated process was developed to clean the data as discussed in the following sections.

Data Cleaning and Pre-processing

In a network, each node represents a unique identity. Hence, the most important task in cleaning the data was to recognise and remove inconsistencies in the names which occurred due to the textual nature of responses. The following techniques were used to clean the data in Python:

• Initially, all the names were made lowercase for the analysis.
• Special characters were removed.
• Rows containing missing or no information were removed.
• Trailing or unnecessary white spaces were removed.
• Incorrect spellings were identified and removed using a spellchecker. The challenge here was to differentiate between the proper nouns (such as ‘EcoZip’) and dictionary words (such as ‘Adventures’). For example, an entity name ‘EcoZip Adventures’ was misspelled as ‘EcoZip Adventres’. A conventional spell checker would consider ‘EcoZip’ as a spelling error along with ‘Adventres’ as both the words are not found in the dictionary. Hence, a solution was developed to distinguish the proper nouns from actual dictionary words in entity names and correct spelling errors in the data.

• A custom algorithm was developed to identify abbreviations in the text and replace it with the full name. e.g. ‘DOC’ was identified as ‘Department of Conservation’.
• Several names which were written similarly but were however, the same entity, were identified and merged. This is the most significant part of the process or the most “satisfying” part, as a user described it. An example is shown below:

• All the names in the network were also compared pairwise to further remove any inconsistencies and generate a list of consistent and unique names involved in the biosecurity information exchange.

This cleaning process reduced the entities from 319 (including inconsistencies) to 139 consistent and unique entities (nodes) in the network with 335 relationships (edges) between them which were then used to generate visualisations.

Social Network Analysis

Directed maps were generated using Gephi which were then further analysed. An anonymised example of one of the social network maps generated is shown below:

An example of anonymised social network map generated using Gephi.

The information obtained through this network analysis can now be used by biosecurity providers to better target information exchange within the New Zealand tourism industry.

Achievements

1. This application significantly reduced the time (from 1-3 weeks to a few hours or minutes) in cleaning and pre-processing the data before analysis.
2. As a result, the costs involved in the conventional extensive processes, which involved a lot of manual effort, were also reduced.
3. The new streamlined process almost eliminated the human errors involved in the manual inspection of data.

Conclusion

Hence, through this case study, an application was developed, which streamlines and automates all the steps starting from loading and cleaning the data up to the generation of data sheets to be used in the SNA. Although, this is currently a Python application, the development of a GUI based interactive SNA application design is currently under consideration.

I would like to acknowledge Helen Percy, my industry supervisor and Penny Payne, the social scientist at AgResearch for their invaluable support during this project.

Author

Romalee Amolic is a Master of Applied Data Science student at the University of Canterbury. In February 2018, she completed her summer internship with AgResearch, Hamilton. She thoroughly enjoyed her internship project which involved streamlining and increasing the efficiency of the data cleaning and network map generation processes at AgResearch. She is passionate about harnessing the power of data analytics to improve the lives of people. She eagerly looks forward to applying the skills learnt, in fulfilling her aspiration of becoming a data scientist.

Romalee (centre) with Helen Percy, her industry supervisor (right), and Penny Payne, the social scientist at AgResearch (left).

In the summer of 2016-2017, Te Hiku Media and Te Pūnaha Matatini co-funded a number of student internships – work from which led to the development of Kōrero Māori – a project to teach machines how to speak te reo Māori.

One of the interns was Jamie Chow, a conjoint BComm/BEng (Honours) degree student from the University of Auckland. Jamie’s work on Te Hiku’s Data Analytics and Visualisation Project involved using online audience data to measure the performance of the organisation’s digital platform, matching it with other information such as demographics and geographical data.

Internship leads to ongoing employment opportunity

Following his 10-week summer internship, Jamie continued working on the project for Te Hiku in part-time employment over the course of 2017.

“We kept Jamie on board,” says Te Hiku’s R&D Scientist and Engineer Keoni Mahelona. “He had the internship then we hired him on contract for about 6 months or so. Just casually, furthering the work that he did for that project. That was really useful.”

Jamie was able to create visualisations of the data that allowed Te Hiku staff to quickly understand and interpret the behaviour of their audience. Staff can now immediately examine their content for audience engagement, allowing them to adapt their future programming.

“The project mainly involved developing tools to obtain usage data for their online platform and present this data in a suitable form to their staff, producers, and content creators,” says Jamie.

“My project was very self-driven so it was always challenging and engaging. It was clear even from the beginning that I would learn a lot throughout the project. Te Hiku Media seemed to be pleased with the results and I’m happy they’re using and benefiting from my work.

“Overall, it was a great experience and I’m grateful that I had the opportunity to work with some awesome people.”

Jamie Chow, Te Pūnaha Matatini 2016-17 summer student intern (centre), with Keoni Mahelona, Te Hiku Media (left) and Shaun Hendy, Te Pūnaha Matatini (right).

About the Kōrero Māori project

Te Hiku’s Data Analytics and Visualisation Project led to discussions about the need for ICT tools for te reo Māori. Working together, Keoni’s team developed the Kōrero Māori project to create computer language models for te reo Māori – the basic tools which will allow machines to be able to transcribe and speak the language.

Keoni is optimistic about how Te Hiku and Te Pūnaha Matatini can collaborate in terms of research direction in the future.

“I do look forward to us continuing to work together… and doing research around Te Reo, language processing and also looking to the languages of the Pacific – because there’s a lot of similarities,” says Keoni. “If we can turn these into machine models, perhaps we can learn something about the evolution of our languages.

Keoni says that Te Pūnaha Matatini has been very helpful and open in terms of how the data are managed.

“It’s about managing data in a way that aligns with our tikanga and our values, as Māori and as a Māori organisation and as indigenous people. Our language is our culture. It’s our identity. We’re talking about giving that to a machine and I guess the question comes, well, who owns that data or who owns the machines that have access to that data,” explains Keoni.

“As indigenous people, we want to maintain some sovereignty… and Te Pūnaha Matatini has been helpful in terms of having quite a broad and open understanding of those aspects of the project.”