Extracting Twitter data by topic, date and top words

[lauespinosa]

🔎📑📅🏷️ Extracting Twitter data by topic, date and top words (epitweetr version 1)

The following function allows epitweetr users to subset their Twitter data by topic, date and top words. Please consider that these data are part of the raw data collected by epitweetr and may differ among users.

Please note that this only applies for epitweetr version 1 and not for the newest version (v2).

Before running the code, epitweetr users must set up the configuration epitweetr::setup_config("wd") where wd is epitweetr working directory selected by the user.

Steps to subset Twitter data:

1. Set up configuration for epitweetr
2. Run the code below to get the function in your R environment
3. Select a topic, year, date and top words for subsetting your Twitter data. For example: signal_tweets('toxoplasmosis', 2020, '2020.10.12', '01', c('toxoplasma', 'gondii'))

signal_tweets <- function(topic, year, date, number, topwords){
  # get working directory
  epitweetr_wd = epitweetr:::conf$data_dir
  
  # topic = topic of interest as included in the column "Topic" in the topics.xlsx
  # year = year of interest
  # date = date of interest (YYYY.MM.DD)
  # number = specific file for that topic and date, in case there are more than one (##)
  # topwords = list of top words of interest
  
  # Stream in the json file of interest
  file <- jsonlite::stream_in(gzfile(paste0(epitweetr_wd, '/tweets/search/', 
                                            topic, '/', year, '/', date, '.000', number, '.json.gz', sep = "")))
  
  # Join top words with OR boolean operator
  topwords = paste(topwords, collapse = "|")
  
  # Look for elements of the tweets list that contains at least one of the top words 
  for (i in 1:nrow(file)) {
    if(nrow(file$statuses[[i]])!=0 && nrow(file$statuses[[i]][grepl(topwords,  
                                                                    file$statuses[[i]]$text, ignore.case = T),])) {
      assign(paste("file_text", i, sep = "_"), file$statuses[[i]][grepl(topwords, file$statuses[[i]]$text, ignore.case = T),])
      
    } else {
      assign(paste("file_text", i, sep = "_"), data.frame(1:2))
    }
  }
  
  # Merge all tweets found
  tweets <<- eval(parse(text=paste("dplyr::bind_rows(", paste("file_text_", 1:nrow(file), 
                                                                    sep = "", collapse = ", "), ")")))

  # Save the file in the Global Environment
  epitweetr_sample_signals <<- tweets[is.na(tweets$text) == F,]$text
  
  # show the text of the subset
  View(epitweetr_sample_signals)
  }

[Abdouzster]

This is a very useful function. If we have another one to extract the tweet or tweets that generated a specific signal (alert), that would be great. Many times when I try to locate such tweets on Twitter, I can't get the tweet. Some of the alerts also does not have keyword to search for.

[Abdouzster]

I have another question: would it be possible to use a similar function to extract tweets about one topic in a specific location?
Thanks,
Abdou