Skip to content

Latest commit

 

History

History
126 lines (83 loc) · 3.76 KB

chapter12.md

File metadata and controls

126 lines (83 loc) · 3.76 KB
theme _class paginate backgroundColor footer marp
gaia
lead
true
Computational Thinking and Social Science | :copyright: Matti Nelimarkka | 2023 | Sage Publishing
true
<style> footer { font-size: small; } </style>

bg left:40% 80%

Chapter 12

Mistakes and Quality of Results in Computational Social Sciences

Learning goals

  • Explain the importance of validity and reliability in scientific research.
  • Identify the validity and reliability concerns that are possible with empirical research contributions,
  • Characterise the major types of software flaws. Assess research questions in terms of the validity and reliability concerns that could arise in research probing them.

Validity and reliability

  • Scholars are interested in identifying the factors that could cause their results to be less credible than they had expected.
  • Internal validity
  • External validity
  • Reliability

Learning activity

Pair or small group discussion: Exercise 12.1.

(Helps if you ask students to list these on whiteboard or flipchart.)

Validity and reliability

  • Computational methods are not immune to traditional concerns.
  • In addition, new concerns emerging from how we work with data and code emerge and require additional attention.

Context and why it matters

  • Most data are produced by humans (either directly or indirectly) and therefore are socially produced and contextual.
  • This influences the shoulds, but there is no limit on the cans. Nothing stops you from making this type of mistakes.

Things to consider:

  • material’s background (i.e., where and when it was created)
  • the social practices shaping the data

Algorithms as black boxes

  • Many algorithms are black boxes: it can be unclear how and why particular outcomes emerge when specific inputs are entered. Especially true when we use libraries produced by others.
  • While libraries increase algorithmic black boxes, we still must use them. We just need to be aware of these issues.

Things to consider:

  • Are you using a library in its intended context?
  • Can you test the face validity of the results?

The bugs infesting everything

  • Software not fulfilling its intended purpose, there are coding mistakes, i.e., bugs.
  • Sometimes event the data has defects we should be aware of and limit what kinds of claims can be made on it.

Things to consider:

  • Clarity and automated testing tools.
  • Explore data and check it makes sense.

The illusion of a ‘standard’ research process

  • In mature fields, there is a 'cookbook' approach for applying methods, but computational methods are so new that a widely accepted 'cookbook' does not exist.
  • Instead, each researcher need to develop their own recipt.

Learning activity

  • Pair or group discussion: Exercise 12.2
  • Pair or group discussion: Exercise 12.4
  • Individual activity: Exercise 12.9
  • Code activity: Exercise 12.10 and 12.11

Mitigation strategies

  • Triangulation: Tools, configurations, replications, method, data
  • Critical reflection: Reflective stance with research methods; why and how computational methods are used?

Learning activity

  • Class or group discussion: Exercise 12.12
  • Individual activity with group discussion: Exercise 12.13 and 12.14

Review questions

  1. Why do scholars seek to understand the limitations of their methods and the errors that could arise from these?
  2. What kinds of mistakes we might make if we do not understand the research context properly?
  3. What new issues emerge when we use algorithms?
  4. What types of bugs may have an impact on the software code?
  5. What strategies are available to mitigate concerns about validity?
  6. How would you acknowledge and address the issues you have found in your own research?