Data: Daily COVID deaths

Overview:

Whom is this course for?

Pre-class preparation

• Bring own device (Windows 10 or MacOS, no tablets)
• All software/applications used in this class are open-sourced
• Programming in cloud platforms (RStudio cloud, Google Colab)
• Recommended accounts: GitHub 

Please fill out this survey                        

This introductory course is an overview of Data Science.  Students will learn:

1. What is Data Science?
2. What is Big Data?
3. How to equip for data scientist
4. Tools for professional data scientists

Prepare for class

Overview:

Why Data Science? Why now?

McKinsey & Co., An Executive’s Guide to AI

Data fluency

Hugo Bowne-Anderson. 2019. "What 300 L&D leaders have learned about building data fluency"

Data fluency

Hugo Bowne-Anderson. 2019. "What 300 L&D leaders have learned about building data fluency"

Everybody has the data skills and literacy to understand and perform data driven documents and tasks

Danger of immature data fluency

Types of Data Science

Rogati AI hierarchy of needs

What is data?

Let the dataset change your mindset.

- Hans Rosling

Hans Rosling

Data Literacy

1. Data generating process
   1. ​How data are generated
   2. Distribution
   3. Missing values
   4. Wrong data

2. Graphic grammar
   1. Bad charts deliver incorrect message
   2. Poor design
   3. Color
   4. Label
   5. Scale

3. Statistical understanding
   1. Size does (not) matter
   2. Representativeness does
   3. Forecast/prediction minded
   4. Explanation

Darkest hour: Churchill and typist

• Data Thinking

• Multi-disciplinary Thinking

• Computationally Thinking

• Programming is a practice of using programming language to design, perform and evaluate tasks using a computer.  These tasks include:
  • Computation
  • Data collection
  • Data management
  • Data visualization
  • Data modeling
• Data Programming with GenAI

What is programming?

Data programming

}

Why learning programming Languages?

- Maribel Fernandez 2014

Compilation

Interpretation

• Machine language
  • Assembly language
  • C

Low-level languages

• BASIC
  • REALbasic
  • Visual Basic
• C++
• Objective-C
  • Mac
• C#
  • Windows
• Java

Systems languages

• Perl
• Tcl
• JavaScript
• Python

Scripting languages

DRY – Don’t Repeat Yourself

Write a function!

Function example

# Create preload function
# Check if a package is installed.
# If yes, load the library
# If no, install package and load the library

preload<-function(x)
{
  x <- as.character(x)
  if (!require(x,character.only=TRUE))
  {
    install.packages(pkgs=x,  repos="http://cran.r-project.org")
    require(x,character.only=TRUE)
  }
}

Why learning programming?

learning how to program can significantly enhance how social scientists can think about their studies, and especially those premised on the collection and analysis of digital data.

   

- Brooker 2019: 

Chances are the language you learn today will quite likely not be the language you'll be using tomorrow.

Beware of bugs in the above code; I have only proved it correct, not tried it."

 

- Donald Knuth, author of The Art of Computer Programming

Data Methods

Made

Data

}

}

Found

Data

Data Methods

Data file formats

YAML (Yet Another Markup Language or YAML Ain't Markup Language) is a data-oriented, human readable language mostly use for configuration files)

Since open data are provided by government agencies or research institutions, these data files are often:

• Structured

• Well documented

• Ready for data/research functions

API

• APIs can take many different forms and be of varying quality and usefulness. 

• RESTful API (Representational State Transfer) is a means of transferring data using web protocols

• Example:
  • Crossref API
    http://api.crossref.org/works/10.1093/nar/gni170
  • Taiwan Legislative Yuan API
    https://www.ly.gov.tw/WebAPI/LegislativeBill.aspx?from=1050201&to=1050531&proposer=&mode=json

Like open data, data available through API are generally:

• Structured

• Somewhat documented

• Not necessary fully open

• Subject to the discretion of data providers

• E.g. Not all variables are available, rules may change without announcements, etc.

For the type of found data not available via API or open access, one can use non-API methods to collect this kind of data.  These methods include scraping, which is to simulate web browsing but through automated scrolling and parsing to collect data. These data are usually non-structured and often times noisy.  Researchers also have little control over data generation process and sampling design.

Non-API data are generally:

• Non-structured

• Noisy

• Undocumented with no  or little information on sampling

Illustration: Collecting stock data 

This workshop demonstrates how to collect stock data using:

1. an R package called quantmod

Illustration: Collecting stock data 

This workshop demonstrates how to collect stock data using:

1. an R package called quantmod

This workshop demonstrates how to collect stock data using:

Link to RStudio Cloud:

https://posit.cloud/content/6625059

- Need a GitHub and RStudio Account

Link to class GitHub:

https://github.com/datageneration/academiasinica

Workshop II: Data collection

Assignment 1

1. Install R and RStudio

2. Download the R program in class GitHub (under datascience/codes)

   1. DPR_stockdata.R

3. Can you download TSM's (台積電） data in the last three years？

4. Plot the TSM data in the last three years using the sample codes (plot and ggplot2)

Illustration: Collecting COVID data 

This workshop demonstrates how to collect COVID data using:

1. API methods
   1. Johns Hopkins University Center for Systems Science and Engineering (CSSE) (map | GitHub)
   2. Our world in Data (website | GitHub)
   3. New York Times (GitHub)

Data: Total cases per million

Data: Daily COVID deaths

Data: Death data (Asia)

Data: Death data (Europe)

Data: COVID cases~predictors

Data: COVID cases~predictors

Assignment 2

1. Download the R program in class GitHub (under datascience/codes)

   1. DPR_coviddata1.R

2. Can you download Taiwan and Germany COVID data in last three years?

3. Plot the data using the sample codes (use plot and ggplot2 functions)

Assignment 3

1. Download the R program in class GitHub (under datascience/codes)

   1. DPR_caret01.R

2. Can you predict the chance of Tsai winning including additional variable "indep" (support for Taiwan's independence")?

3. What is the new accuracy?  Better or worse? 

Assignment 4 (optional for AP)

1. Download the R program in class GitHub  (under codes)

   1. DPR_tuber01.R

2. Can you download channel and video data from “中天新聞” and “關鍵時刻”?

3. Can you create WordClouds for selected video from each channel?

Illustration: Scraping YouTube data 

This workshop demonstrates how to collect YouTube data using:

1. API method (with Google developer account)

Illustration: Collecting YouTube data 

This workshop demonstrates how to collect YouTube data using Google API:

Link to RStudio Cloud:

https://rstudio.cloud/project/4631380

- Need a GitHub and RStudio Account

Link to class GitHub:

https://github.com/datageneration/nchu

Illustration: Scraping Twitter data (deprecated)

This workshop demonstrates how to collect Twitter data using:

1. API method (with Twitter developer account)
2. Non-API method (using Python-based twint)

• Notebook